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Introduction 

Let (X, 0) be a principally polarized abelian variety (ppav) over C of dimen- 
sion g > 2 with a symmetric theta divisor 0. The intersections 

y x = e.ne^, 

where Q x = + x denotes the translate of by x G X(C), have been studied 
in relation with Torelli's theorem [Debl] and the Schottky problem |Deb3j . 
We examine the variation of Q-Hodge structure on H'(Y X ,Q) for x G X(C) 
varying. For example, if is smooth, Y x is smooth of dimension g — 2 for 
generic x G X(C). Then by the weak Lefschetz theorem the cohomology of 
Y x is obtained by restriction from the cohomology of X except for the middle 
cohomology degree. So we only need to consider the quotient 

H = H°- 2 (Y X ,Q)/H°- 2 (X,Q). 

The involution o = —idx acts on Y x and thereby induces an eigenspace decom- 
position H = H + © H_. The eigenspaces H± are the fibres of two variations V± 
of Q- Hodge structures which we study in section [TJ 

Conjecture A. // smooth, then V± are simple. 

Viewed as Hodge modules in the sense of |Sa2| . the variations V± of Hodge 
structures have two underlying perverse sheaves 5± on X, see section El Thus 
conjecture IA1 would follow from 

Conjecture B. If Q is smooth, the perverse sheaves 5± are simple. 

Up to a skyscraper sheaf, we construct 5± in section 13.41 as the symmetric 
resp. alternating square of the perverse intersection cohomology sheaf 5® in 
the tensor category introduced in jWe2j. This category is equivalent to the 
category Rep(G) of algebraic representations of some (in general unknown) 
algebraic group G = G(X, 0). So conjecture IB1 would be a consequence of 
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Conjecture C. If is smooth, then 
G{X,Q) = 



SO(g\,C) for g odd, 
Sp(g!,C) for g even, 



and 5q corresponds to the standard representation of this group. 

A motivation for this is given in section H~3l In fact, via an analysis of small 
representations [KrWl| and a Mackey argument, we will see in section [TT1 that 
conjectures [B] and [C] are equivalent. 

Since every ppav of dimension g < 3 with a smooth theta divisor is a Jacobian, 
the above conjectures are true in these cases |Wel] . The first new case appears 
for (7 = 4. If we drop the assumption that G is smooth, this is also the first 
non-trivial case for the Schottky problem (viz. to characterize the Jacobians 
among all ppav's). In section [5] we discuss the relationship between the Schottky 
problem and our conjectures in this case. 

The main result of this paper is the proof of conjecture C for g = 4. In 
the proof given in sections [9] and [10] we consider a degeneration of (X, G) into 
the Jacobian variety of a generic curve of genus 4. This provides a restriction 
functor p : Rep(G(X, 0)) — > Rep(G^,), where G^ C G(X, 6) is an algebraic 
subgroup defined via the formalism of the nearby cycles \l/ as outlined in sec- 
tion O To prove the conjecture we compare p with another functor MT from 
Rep(G(X, 0)) to the category of representations of the Mumford-Tate group of 
B. For the study of this second functor the main input will be the knowledge 
of the primitive cohomology of which we express in section [2] in terms of 
the intermediate Jacobian of a cubic threefold as in |Do] and [Iz], using results 
of [£!q], [CM] and |EFS) . 



1. Variations of Hodge structures (conjecture |A1 

In this section we assume that is smooth, and we study the variations V± 
of Q-Hodge structures and their Hodge decomposition for g < 4. 

1.1. Smoothness of Y x . For generic x G X(C) the translates Q x and Q- x 
intersect each other transversely and hence Y x is smooth. 

Proof. is defined by the zero locus of the Riemann theta function 6(z) = 
8(t, z) on the universal covering p : C 9 — > <C 9 / (Z 9 + rZ 9 ) = X(C). For smooth 
the gradient 9'{z) is non-zero for all z G C 9 with 8(z) =0. If our claim were 
false, we could find a non-empty analytic open subset V G C 9 and a complex 
analytic map s : V — > C 9 such that p ( 2 ) and 0- p ( 2 ) intersect non-transversely 
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in s(z) for all z G V, i.e. such that 

(1.1.1) 6(s(z) + z) = 9(s(z)-z) = 0, 

(1.1.2) 0'{s{z) + z) = X(z) ■ 9'(s(z) - z) for some X(z) G C*, 

for all z G V. With the g x g unit matrix E and the Jacobian matrix (Ds)(z) 
of s at z, taking gradients of ( ll.l.ip implies 

(1.1.3) (E + (Ds)(z))-6'(s(z)+z) = 0, 

(1.1.4) (E — (Ds)(z)) ■ 0\s(z) — z) = 0. 

Now multiply (11.1.41) by A (z). If we plug in (ll.l.2p and add ( ll.l.3p . we get 

8'(s(z) + z) = contradicting the smoothness of 0. □ 

1.2. Variation in a family. Over some Zariski-open dense U C X there exists 
a smooth proper family tt : Yjj — >• U with fibres ir~ 1 (2x) = Y x and an etale 
involution a :Yjj —^Yjj with ct^-im^ = (—idx)\Y x f or ^ x ^ n U(C). 

To construct such a family, let tt : 0x0 — » X be the addition map. Projecting 
from 7r _1 (2x) C x onto the first factor and then translating by x, we get 
an isomorphism tp : 7r _1 (2a;) = X D 0-^ such that via (p the involution a : 
0x0 — > 0x0, (ti,t 2 ) i — y (t 2 , ti) becomes identified with {—id x )\e x ne- x - By ll.ll 
the fibres of tt are generically smooth. So there is an open dense U C X such 
that for Yjj = 7r _1 (f7) the restriction ti = tt\y v is smooth, and for sufficiently 
small U the involution a will be etale on Yjj. 

1.3. Definition of V±. For all u, the higher direct images R^tt^Qyu) are vari- 
ations of Q-Hodge structures \PS\ cor. 10.32]. It is easy to construct a constant 
subvariation H v R v tt*{Q.y u ) such that at every point 2x G U(C), the fibre 
^ v C R u tt*(Q Yu )2x = H U {Y X , Q) is the pull-back of the cohomology of X to Y x . 
In particular H u = H U (Y X , Q) for aRu^g-2. We put 

V = R?- 2 ir.(Qy v )/H°- 2 

and define V± as the eigenspace of a* for the eigenvalue ±(— l) 9 ^ 1 , i.e. 

V± = ker( ( x*T(-l) 3 - 1 ^y). 

The reason for this choice of signs will become clear later (see 13. 4p . 

1.4. Small-dimensional cases. In this section we fix a point x in X(C) such 
that 2x G [/(C). We use the notation Y = Y x , and we define Y + as the quotient 
Y/(a) of y by the involution a = — idx\y- 

The case g = 2. Here Y consists of 2 points, Y + is a single point, V_ = 
and V+ = Q;/ is the constant variation with Hodge degree (0, 0). 

The case g = 3. This case has been studied in [Re], |Krl] . Here (X, 0) is 
the Jacobian of a smooth curve C and Y" + — >• Y is an etale double cover of 
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smooth curves with Y of genus 7. Now to any etale double cover of curves 
one may associate a ppav, its Prym variety |Mul] |BLl ch. 12]. It turns out 
that for generic (X, 0) the Prym variety P of the cover Y — > Y + is isomorphic 
to (X, 0). Every etale double cover with this Prym variety arises like this 
for some x. Furthermore, the coverings for two points X\,Xi are isomorphic 
iff X\ = ±X2- Hence the etale double covers with given Prym variety (X, 0) 
are parametrized by an open dense subset W of the Kummer variety X/(±l). 
Points outside W parametrize degenerate double covers. 

By construction of Prym varieties, the Jacobian variety JY is isogenous to 
P x JY + , hence H\Y, C) S H\X, C) © H\Y + , C). Thus V_ = 0, and V+ is 
a variation of Hodge structures of abelian type whose fibres H 1 (Y + , Q) = Q 8 
have Hodge degrees (1,0) and (0, 1). 

The case g = 4. Here Y — > Y + is an etale covering of smooth surfaces. We 
claim that V+ has rank 52, with Hodge degrees (2, 0) and (0, 2) of rank 11 and 
Hodge degree (1,1) of rank 30, whereas V_ is a variation of Hodge structures 
of rank 6 purely concentrated in Hodge degree (1, 1). 

In particular, by the Lefschetz (1, l)-theorem the fibers of V_ are spanned by 
six cycles on Y + which generate H 1 ' 1 (Y + , C)/iJ 1 ' 1 (X, C). So V_ has an under- 
lying finite monodromy group T = T(X, 0) with a six-dimensional faithful rep- 
resentation. Computing H 2 (X,/L) and H 2 (Y + ,Z) and using a classification of 
lattices with small discriminant, one sees that the fibres of V_ have the underly- 
ing Neron-Severi lattice Eq(—1), so F is a subgroup of Aut(E§) = W(Eq) x {±1}. 
From the intersection configuration of the 27 Prym-embedded curves in Y of [Iz, 
sect. 4.3], one then deduces that the projection W(Eq) x {±1} — > W(Eq) maps 
T isomorphically onto a subgroup of W(Eq). To get a lower bound on T consider 
a degeneration of (X, 0) into a Jacobian variety (X , O ). Let T = T(Xq, Qq) 
be the monodromy group underlying the analog of V_ on (X , O ), then r is a 
subquotient of V. In the Jacobian case the configuration of the 27 Prym curves 
of loc. cit. is no longer symmetric; precisely 12 of them are smooth. They come 
in 6 pairs of curves which are interchanged by the involution ±idx , and the 
associated 6 cycles are permuted by the monodromy operation of IV From this 
one deduces that To contains the alternating group A§. Altogether one then 
concludes that T must be the Weyl group W(Eq) or its finite simple subgroup 
of index 2. For the details see |Kr2] . 

Proof of the claim about the Hodge decomposition ofV±. The Hodge numbers 
of the fibres of V± are h p > q (Y) - h p,q (Y + ) resp. h M (Y + ) - h M (X) for p + q = 2, 
so it suffices to check the following table: 





h 2,0 = h 0,2 


h 1 * 1 


h i,o = h o,i 


Y 


17 


52 


4 


Y+ 


6 


22 
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The last column of this table follows from if^Q) = H^X^Q). To check 
the remaining two columns, put C x = Ox{Qx)- Since Tx = {Ox)® 4 and hence 
CtiTxW) = 1, the exact sequences — >Tq x \y — >Tx\y— >C x \y — ^0 and 0— )• 7y— > 
TqJy -> £-x\y -> Q snow 1 = c t(7y)(l + ci(C- x \ Y )t)(l + Ci(£ x \ Y )t). Since £ x 
and are numerically equivalent to C = (9x(0), this gives c 2 (7y) = 3cf(£\ Y ). 
Then xCD = deg y c 2 {T Y ) = 3 deg y c 2 {C)\ Y = 3 deg x cf(£) = 3 • 4! = 72 by the 
Gauss-Bonnet and Poincare formulae. It follows that h 2 (Y) = 86 since h°(Y) = 
h (Y) = 1 and h l {Y) = h 3 (Y) = 8 by the hard Lefschetz theorem. Furthermore, 
from Ci(7y) = — 2ci(£|y) we get deg y c^(7y) = 4deg y cf(£)|y = 96, hence 
1 - 4 + h°' 2 (Y) = x{Oy) = (c?(7y) + c 2 (7y))/12 = 14 by Hirzebruch-Riemann- 
Roch. So h°' 2 (Y) = 17. Finally, since Y — > Y + is an etale double cover, 
X (Y+) = x(F)/2 = 36 and x{Oy+) = x(O y )/2 = 7. Using that h°(Y+) = 
h A (Y + ) = 1 and h l (Y + ) = h 3 {Y + ) = since a acts by (-l) fc on H k {X,C), it 
follows that h 2 (Y + ) = 34 and h°' 2 (Y + ) = 6. □ 

2. The Mumford-Tate group MT(6) in Genus 4 

One of the key ingredients to the proof of conjecture iBl for g — 4 will be 
the analysis of the Hodge structure on tensor products of M*(X, 5q). This is 
encoded in the Mumford-Tate group MT(0) as we recall below. For reductive 
groups G let G sc be the simply connected covering of the derived group [G°, G°] 
of its Zariski connected component G°. We say, a subset of an algebraic variety 
is meager if it is contained in a countable union of proper closed subvarieties. 
The goal of this section is to prove the following 

Theorem. For every ppav (X, 6) outside some meager subset of A4, the theta 
divisor satisfies 

MT(9) SC = MT(Jf) x Sp(10, C) where MT(X) = Sp(8, C), 
so we have canonical homomorphisms 

MT(X) MT(6) SC Sp(10, C) 




MT(6) 



If we consider V = M*(X, 5q) as a representation o/MT(0), the induced action 
o/MT(X) on V describes the part of the cohomology which comes by restriction 
from X. The action o/Sp(10,C) describes the remaining part. 

Recall [DelTl sect. 2.1], that giving a Q-Hodge structure is tantamount to 
giving a finite-dimensional vector space V over Q and a homomorphism h : 
S = Resc/R(G mj c) — > G1(V)jr of real algebraic groups whose composite with the 
weight cocharacter of S is defined over Q. The Mumford-Tate group MT(V) 
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is defined as the smallest algebraic subgroup of G1(V) over Q through which 
h factors. The Q-Hodge substructures of any tensor power of V are precisely 
its MT(V)-stable subspaces. The Mumford-Tate group of polarized Q-Hodge 
structures is reductive |Del4[ prop. 3.6]; this in particular applies to MT(X) := 
MT(H'(X,Q)) for smooth projective varieties X over C. 

2.1. The Hodge structure B. Let A* denote the i-th exterior power of the 
standard representation of Sp(2g, Q). For every ppav (X, 0) outside a meager 
subset of A g then by |BL| prop. 17.3.2] 

MT(Jf) = MT(X) SC = Sp(2#,Q) and iT(X,Q)=A l . 

If is smooth, the weak Lefschetz theorem shows that for some Q-Hodge 
structure B and < |z| < g — 1 

' As- 1 -!*! f or i ^ o, 
A 9 - 1 © B for i = 0, 



in particular MT(0) = MT(#*(X,Q) © B). Now it is well-known that for 
any Q-Hodge structures V±, V2 and V — V\ © V2 we have a closed embedding 
1 : MT(V) «^ MT(V r i) x MT(V r 2 ). The image of 1 surjects onto each of the two 
factors. If the Mumford-Tate groups are reductive, this surjectivity carries over 
to MT(— ) sc , and the kernel of the induced map i sc : MT(V) SC — > MT(Vi) sc x 
MT(V2) SC is contained in the kernel of the natural map MT(y) sc — > MT(V). 
Hence we have a commutative diagram 

MT(0) sc 





Sp(2<7,Q) Sp(2^,Q) x MT(B) SC MT(B) SC 

with kei(i sc ) C ker(MT(0) sc MT(0)). 

Lemma. To prove the theorem, it suffices to show that for every ppav (X, 0) 
outside a meager subset of Aa, the Hodge structure B is simple. 



Proof. Indeed, the simplicity of B is equivalent to EndMT(B) {B) = Q. By \Ri\ 
th. 1] this is equivalent to MT(B) = MT(B) SC = Sp(10,C), since B is a Hodge 
structure of abelian type (see section l2~2l) . 

Now the projection MT(0) sc -» MT(X) = Sp(8,C) shows that there is a 
reductive group G with MT(0) sc = Sp(8, C) x G. If B is simple, it follows 
that B = B\M B2 with irreducible representations B\ of Sp(8, C) and B2 of G. 
If B\ were non-trivial, dim(S) = 10 would imply dim(5i) G {2,5,10} which 
is impossible |AEV] . Hence Bi is the trivial representation, B 2 is the standard 
representation of MT(B) = Sp(10,C), and one easily deduces that l sc is an 
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isomorphism. □ 

The rest of this section is devoted to the proof of the theorem. By the last 
lemma it suffices to show that for every ppav (X, 0) outside a meager subset 
of A4 the Q-Hodge structure B is simple. Notice that MT(B) C Sp(10, C) in any 
case |Go[ B.62], with equality holding iff B is simple. Since the Mumford-Tate 
group of a variation of Hodge structures is constant outside the complement of 
some meager subset and only becomes smaller on this meager subset, it therefore 
suffices to prove the simplicity of B for a single ppav (X, 0) in A\. 

2.2. Intermediate Jacobians of cubic threefolds. For any smooth cubic 
threefold TcPj one has the intermediate Jacobian 

JT = H 2 > l (T)*/H 3 (T,Z) 

which by |CG] is a simple ppav of dimension 5 and determines T uniquely up to 
isomorphism. Let T C A5 be the closure of the locus of all these intermediate 
Jacobians. In |Do| and [Iz] a smooth cubic threefold T = Trx,e) has been 
associated to each (X, 0) in a Zariski-open dense subset A\ C A±. This gives a 
map (p : Al — > T which is generically finite and dominant by |Dot birationality 
of x in thm. 5.2 (2)]. By Chevalley's theorem the image of ip contains a Zariski- 
open dense subset of T ■ 

For (X, 0) G A\ and for the corresponding Q-Hodge structure B as in 12. 1} 
the arguments of |IvSj imply 

B = H\jT {x ,e),Q)(-l). 

If for abelian varieties A we put End (A) = End(A) cg>^ Q, a basic property of 
Mumford-Tate groups |Got B.60] hence shows 

End U T(B)(B) = End°(JT (x , e) ). 

Thus B is a simple Q-Hodge structure if and only if End°(JT(x,e)) = Q. In what 
follows we prove the latter property for suitable (X, 0) in A\ by a degeneration 
argument. Using a Collino family, this will be reduced to the study of an 
extension of a generic Jacobian variety by a torus. 

2.3. Collino's family. As in |Cot part II] and |CMt p. 44-45] one finds a 
group scheme B —> S over a non-empty Zariski-open subset S'cPj. and a point 
s G S(C) such that 

• for all t ^ s one has B t = JT(x t ,o t ) f° r some (X t , Q t ) G A° Al 

• the special fibre B s is an extension 

E : — ► G m — ► B s — ► JC — > 

for some general curve C of genus 4. As recalled in lemma I5TTI below, the 
theta divisor of its Jacobian JC has two singular points ±e. The class 
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of the extension E in Ext(JC,G m ) is mapped under the isomorphism 
Ext{JC,G m ) ^ Pic°(JC) ^ JC [Si 16.VII] to the point 2e or -2e 
depending on the choice of e. Since C is general, we know that JC is not 
in 9 nu u 4. Hence e 7^ — e by lemma I5~lj so the extension E is non-trivial. 

To prove our theorem we will show End°(B t ) = Q for all t outside a meager 
subset of S(C). Over S* = S \ {s} the family B — > 5 restricts to an abelian 
scheme £>* — >■ 5*. Hence by |Del2l prop. 7.5] and [Dellj 4.1.3.2] the restriction 
map End°(B*/S*) — > End°(B t ) is an isomorphism for all but countably many 
t G 5*(C) (this is why we had to exclude a meager subset of ppav's in the 
theorem). So it suffices to show End(B* / S*) = Z, and this is equivalent to the 
claim in 12.61 below. 

2.4. Endomorphisms of the special fibre. We first show that for general 
choice of C one has 

End(B s ) = Z. 

Proof. Every ip G End(B s ) preserves the toric part G m C B s and induces an 
endomorphism ipjc of JC = B s /G m . So we have a ring homomorphism 

(-)jc: End(B s ) — > End(JC), if; ^ ifj jc . 

For general C we know End(JC) = Z, and then {—)jc is surjective because its 
image contains 1 = (id^Jjc- Now suppose if) G End{B s ) and -^jc = 0, i.e. if) 
factors over G m C B s . Then is a character 2 ^ z" of G m for some n G Z. 
If ?/> 7^ 0, we must have n^O. However, the image of the restriction map 

res: Hom(B s ,G m ) — > Hom(G m ,G m ) = Z 

contains n = res(ip), so the image of the differential d of the .Erf-sequence 

• ■ • Z = Hom(G m , G m ) — —> Ext(JC, G m ) — > ■ ■ ■ 

is a quotient of Z/nZ. In particular, the image of d is an n-torsion group. 
But by construction of the -Erf-sequence, d(id& m ) is the class of the extension 
defining B s . Hence ±2e 7^ is an n-torsion point of JC. For C varying this 
contradicts lemma [2751 below. Hence (—)jc is an isomorphism. □ 

2.5. Torsion points. Let AI4 be the moduli space of smooth curves of genus 
4. Over some Zariski-open dense subset U of Ai^, the map C h-> ±2e defines a 
section of the universal Jacobian variety, and this section is not the zero section 

o w . 

Lemma. IfU C .M4 is a Zariski-open dense subset and ^ On is a section of 
the universal Jacobian variety ir : X — > U, then S defines a non-torsion point 
in all fibres of ix over the complement of a meager subset ofU. 
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Proof. For m G Z, let Z m denote the zero locus of m ■ E. Then i? m is a 
Zariski-closed subset of U, and we must show it is not all of U. If it were for 
some m > 0, then for minimal such m the section £ would define a section over 
some Zariski-open dense subset of Ai^ to the map m M.i — > where m A^4 
is the moduli space of cyclic etale covers of precise order m of curves of genus 
4. Then m M.^ would be reducible, contradicting [BF|. □ 

2.6. Endomorphisms of the generic fibre. For the generic point r\ of S we 
now claim 

End{B v ) = Z. 

Proof. Let Af be the Neron model TV of B v over S. Its universal property 
gives an S'-morphism B — > M . Since this morphism induces an isomorphism of 
the generic fibre and since B So is semi-abelian, by [BLR, prop. 7.4.3] it induces 
an isomorphism of B onto the connected component A/" C N '. In section I2T41 we 
have shown End(B SQ ) = Z. Now consider the composite ring homomorphism 

(p : End°{B v ) ^ End°{Af v ) ^ End\N / S) ^ End°{B s ) = Q , 

where the first isomorphism comes from the identification B v —> Af v , the second 
isomorphism is due to the universal property of Af, and the third map res de- 
notes restriction to the fibre B s = (Af°) s . The image of ip contains 1 = (f(idjs ), 
so if is surjective. On the other hand, B v is a simple abelian variety; indeed, 
all Bt with t ^ s are intermediate Jacobians of smooth cubic threefolds, hence 
simple by |CG| . Therefore End°(B rj ) is a skew field, and since cp is a surjective 
ring homomorphism, it follows that ker(</?) = 0. Thus End°(B v ) = Q, and our 
claim follows. □ 



2.7. Higher genus. For the rest of this section we drop our assumption that 
(7 = 4. It seems likely that for general (X, 0) the cohomology 

decomposes into at least L^y^J simple Q-Hodge substructures. To illustrate this 
let us consider the case of Jacobians. Let C be a general curve of genus g > 3. 
For v G Z let K v be the z/-th exterior power of the standard representation of 
MT(Bjc) sc = Sp(2g,Q), and consider H t/ ( JC, Sq jc ) as a representation of the 
group (a*) x MT(<djc)sc for the involution a = —idjc- Let sgn be the nontrivial 
character of (er*). 

Lemma. For \u\ < g — 1 define n(v) = g — 1 — \u\. Then 

m u (JC,5 ejc ) = sgn n{v)+fM K A n M~ 2 » . 
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Proof. Up to the a-action this is clear as W(JC,6e JC ) = A 9_1 (H'( JC, 8 C )) 
by the non-hyperelliptic case of [Well cor. 13(iii) on p. 64 and p. 124], but 
beware that unlike Sq jc , the perverse sheaf 5c is not a-equivariant! To find the 
cr-action on H*( JC, Sq jc ), we use the evaluation map |BrB[ sect. 4.2] 

H'(JC,Q)® Q Q[x] — ► H'iC^^Q) W(JC,6 ejo )[l-g] 

which is an isomorphism in degrees < g — 1 and a-equivariant, where on the 
left hand side o acts in the usual way on H'(JC, Q) and on the powers of the 
variable x it acts by 



as observed in prop. 4.3.1 of loc. cit. From this our claim easily follows. □ 

3. Convolutions of perverse sheaves (conjecture ED 

In this section we introduce the perverse sheaves 8± that are used in the 
formulation of conjecture [B] 

3.1. Convolution. Put k = C or k = Q t for a fixed prime I, and denote 
by Dc{X, k) the triangulated category of bounded constructible complexes of 
sheaves with coefficients in k as in [KWJ. Using the addition a : X x X — > X, 
define the convolution of 71, 72 G D C {X, k) to be 71 * 72 = i?a* (71 M 72). With 
respect to convolution Dc(X,k) is a fc-linear tensor category |Wel| sect. 2.1] 
whose unit object is the skyscraper sheaf 5{o} — 1 supported in the origin. 
For hypercohomology H°(X, 71 * 72) = H"(X, 71) ® M*(X, 72) by the relative 
Kiinneth formula, so W(X, — ) is a Cgxfunctor on (D C (X, k),*). 

3.2. Let Perv(X) C D^(X, k) be the abelian category of perverse sheaves on X 
[K\\ I . For a closed subvariety z : Z >■ X and a smooth open dense j : 
U ^ Z, let 5 Z = lC z [d] = i*jUku[d]) e Perv(X) for d = dim(Z) be the 
perverse intersection cohomology sheaf of Z, and put X z = i*j*(ku[d]). If Z is 
normal, X z = k z [d] by |Well lemma 1, p. 5]. Both 5 Z and X z are of geometric 
origin [BBD, sect. 6.2.4] and only depend on Z but not on the choice of U. 
Thus for smooth Z we get 5 Z = X z = k z [d}. 

3.3. The semisimple tensor category P(X). It has been shown in [Wei, 
cor. 6, p. 36] that every simple perverse sheaf 5 G Perv(X) of geometric 
origin with M*(X,5) 7^ is a rigid object in (7J C (X, k) , *) with dual object 
<5 V = (—idx)*{D(5)), where D denotes the Verdier dual. Using this we want 
to construct as in |We2] a semisimple fc-linear rigid abelian tensor category 
P(X) whose objects are represented by semisimple perverse sheaves. Note that 
the convolution of perverse sheaves is in general not perverse. 
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By [KrW2j the perverse sheaves 5 G Perv(X) with Euler characteristic x{$) = 
define a thick subcategory T{X) C D^{X, k), and a perverse sheaf is in T(X) 
if and only if any of its constituents is in T(X). Clearly T(X) defines a Cg>-ideal 
in Dc(X,k), so the quotient category D^(X,k) = Dc(X, k)/T(X) is again a 
fc-linear tensor category. All simple objects in it are rigid. 

Although the full abelian subcategory P(X) C Dc(X, k) of semisimple per- 
verse sheaves on X is not stable under convolution, it turns out in loc. cit. 
that its image P(X) in D^(X, k) indeed is, which in terms of [We2] amounts 
to saying that every (semisimple) perverse sheaf on X is a multiplier. So P(X) 
is a fc-linear semisimple rigid abelian tensor category under convolution. Sim- 
ilarly, via [Well lemma 10, p. 36] the mixed perverse sheaves define a fc-linear 
rigid abelian tensor category P m ixed{X) under convolution, and this category 
contains P(X) as a full subcategory. 

3.4. Definition of 5±. Assume is normal and hence irreducible |EL| . The 
convolution square 5q*5q contains the unit object 1 precisely once, since Sq is a 
simple self-dual object of P(X). Furthermore the commutativity constraint S : 
5 e *5 e = 5 e *5 e of [Well sect. 2.1] is multiplication by (-1) 9-1 on H°(5 e *5 e ) : 
Indeed, one has 5q = ICe[<? — 1]. The commutativity constraint for ICe is 
the identity on T-L 29 ~ 2 (1C@ * ICe)o- This follows from considering fundamental 
classes, since we may replace ICe by the constant sheaf as is normal. The 
shift by g — 1 accounts for the factor ( — l) 3-1 . 

This being said, it follows that 1 lies in the alternating square A 2 (#e) for even 
g and in the symmetric square S 2 (5@) for odd g. So there are perverse sheaves 
5± without constituents from T(X) and complexes r± G T{X) such that 



If is smooth, our construction of the family 7T : Yy — > U in sect ion [L2l shows 
that i?7r*(/cy)[2(7 — 2] corresponds to (5q * Sq)\u- To prove our claim from the 
introduction that conjecture IB1 implies conjecture lAl let us check that V±[2g — 2], 
as a Hodge modules in the sense of [SalJ and |Sa2] . have the underlying perverse 
sheaves 5±\u- 

Indeed, the commutativity constraint is S — Ra*((f>) for the involution <fi of 
k e [g - 1] H k Q [g - 1] given by <j>(s mt) = (-1) 9 " 1 • t K s. Thus S\u is the a of 
section [L2l on R7i^(kY v ) twisted by (— l) 9 ^ 1 , and S 2 (6@)\u and A 2 (5e)|c/ are the 
part of RTT^lky^j) on which a acts by ±(— 1) 9_1 respectively. It only remains to 
notice that r± in lemma 13.51 are the constant subvariations of section 11.31 as we 
will check now. 




and 
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3.5. The translation-invariant summands r± in 5q *5o can be computed explic- 
itly as follows. 

Lemma. // is smooth, then 

r± = HV- 2 -^(X,k)®5 x [fi}. 

fi odd for "+ '' 
/j, even for "—" 

Proof. By semisimplicity we have a decomposition t± = t'±@ t± where t'± 
denote the direct sum of all complex shifts of 5x that enter r' ± . In particular 
then W(X,t±) = since every translation-invariant simple perverse sheaf dif- 
ferent from 5x has vanishing hypercohomology |Wel[ sect. 2. 3]. Hence t' ± can 
be computed from hypercohomology as follows: 

Using the Klinneth formula W(X, 5 e * S e ) = W(X, S e ) ® W(X, 8 e ) and the 
fact that M. U (X,5±) = for v < —g by lemma H'2.2\ one then checks that in 
perverse cohomology degrees fi < the complexes r' ± coincide with the right 
hand side of the lemma. By the hard Lefschetz theorem the same then also 
holds in perverse cohomology degrees /i > 0. Finally, using the result for t± for 
fi = 0, we have the stalk cohomology 

U- 9 (t' ± ) = H°- 2 (X, k) = H°- 2 (Q, k) = U- 9 (6 @ * 5e)o = H~ 9 {r' ± © <) , 

so the non-constant translation-invariant complexes t± must be zero. □ 

4. TANNAKIAN CATEGORIES (CONJECTURE |Cl) 

We now discuss the construction the algebraic group G(X, 0) mentioned in 
conjecture [Cj again following [We2j and [KrW2j. 

4.1. Definition of G(X, 0). Suppose is normal. Inside the rigid abelian 
tensor category P(X) of section 13. 3[ we consider the full abelian tensor sub- 
category P(X, 0) = (5q) generated by Sq. By |KrW2j there exists an affine 
algebraic group G = G(X, 0) over k together with an equivalence 

u: P(X,e) ^ Rep(G) 

of tensor categories, where Rep(G) denotes the tensor category of algebraic 
representations of G over k. Since 5q is a tensor generator of P(X, 0), the 
action of G(X, 0) on u(5 e ) isjaithful. Let P(X, 0) C D b c (X, k) denote the full 
monoidal subcategory above P(X, 0). 

4.2. For objects 7 G P(X, 0) consider the Euler characteristic x(t) = 2~2i<=z(~ 1)' dimfc(ff(X, 7)). 
We claim 

dim fe (w(7)) = x(l)- 
Indeed, the composite cp : 1 ^> 7 * 7 V = 7 V * 7 1 of coevaluation 

and evaluation in P(X,@) satisfies co(ip) = d ■ id for d = dim/ c (a;(7)). Hence 
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M*(X,ip) = d ■ id, and since H*(<£>) is also the composite of coevaluation and 
evaluation in the category of super vector spaces, we get d = xil)- 

4.3. On the representation-theoretic side, the fact that the unit object 1 enters 
A 2 (5q) resp. S 2 (5q) means that the representation u(Sq) respects a symplectic 
resp. orthogonal bilinear form. So we have the following 

Lemma. If is smooth, then 



Proof. By the dimension formula 14. 2\ a Gauss-Bonnet calculation like the 
one in section [TT41 shows dimfe(o;(5e)) = <?! provided is smooth. Furthermore, 
in section 13.41 we have seen that 1 occurs in S 2 (Sq) resp. A 2 (5@) for g odd 
resp. even. This proves the lemma with 0(g!,C) in place of SO(g!,C). So it 
only remains to note that by |KrW2] the group G(X, 0) does not admit non- 
trivial characters. □ 

This lemma and the cases g — 2, 3, 4 motivate conjecture O For g = 2 every 
ppav (X, 0) with smooth is the Jacobian of a hyperelliptic curve; this case 
is covered by |WeH p. 124 and th. 14]. For g = 3 every ppav (X, 0) with 
smooth is the Jacobian of a non- hyperelliptic curve, and o;(5e) is the second 
fundamental representation of G(X, 0) = Sl(4, C)///2 by loc. cit. It corresponds 
to the standard representation of SO(6,C) via the exceptional isomorphism 



We now relate conjecture O to the Schottky problem. For this we drop the 
assumption that is smooth, expecting that the ppav's which are Jacobians 
of curves are detected by the corresponding G(X, 0). Indeed, for genus g = 4 
this turns out to be true as we show in this section. 

5.1. Stratification of the moduli space. Each (X, 0) defines a point in the 
moduli space A g of ppav's of dimension g. The locus Af g C A g of ppav's with 
singular theta divisor is a divisor which for g > 4 has precisely two irreducible 
components [Deb2j. The first component is the locus Q n u\\,g of all ppav's that 
admit a symmetric theta divisor containing a 2-division point with even multi- 
plicity; the second component contains the closure J g of the locus of Jacobian 
varieties and for g = 4 is equal to it [Be] . Let J gt h yp and A g ^ec be the closure of 
the locus of hyperelliptic Jacobians resp. of decomposable ppav's in A g . Notice 
that A^dec c nu u^ n 3t±. 




Sl(4, C)//x 2 = SO(6,C). 



5. The Schottky problem in genus 4 
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Lemma. For g — 4 the following holds: a) On A4 \ A/4 the theta divisors are 
smooth, b) On J 4 \ [9 nu n4 H J4) they have 'precisely two singularities, both 
ordinary double points, c) On 9 nu u 4 \ (9 null 4 H J4) they have precisely one 
singularity, an ordinary double point, d) On (9 nu u 4 D J 4) \ (Ji^yp U A^dec) 
they have precisely one singularity, but the Hesse matrix of the Riemann theta 
function has rank three there, e) On J^h yp \ {^hyp^A^dec) their singular locus 
is of dimension one. f) On A^dec it is °f dimension two. 

Proof. It has been shown in [GSM, thm. 10 and cor. 15] that for b) - d) 
the rank of the Hesse matrices is the given one. In particular, for b) and c) we 
only have isolated singularities. This also holds for d) by Brill-Noether theory. 
The number of these isolated singularities can be obtained from the description 
in |CvdG[ sect. 10] and the generic results of |BeJ and |Mu2[ case b2, p. 55f]. 
Part e) follows from Brill-Noether theory, and f) is trivial. □ 

5.2. The Schottky problem in genus 4. As an application of our results we 
claim that 

G(X, 0) determines the loci J4 and 9 nu u 4 in A4. 

Indeed, for (X, 6) in A 4 \ A/4 we have G(X, 0) = Sp(24, C) by conjecture [C] 
which will be proven for g = 4 in sections [9] and [10] below. By way of contrast, 
for ( X, 0) in J4 \ (J4,h yp U A,d ec ) resp. in J^ hw \ {M,dec n J^hyp) we know 
from [Well p. 124] that G(X, 9) is Sl(6,C)//i 3 resp. Sp(6,C). Finally, for 
(X, 0) in 9 nu u A \ (J4 fl 9 nu ii^) we claim that G = G(X,Q) is different from 
the groups occuring above: From remark [10.31 one gets x{$e) — 22 for (X, 0) 
in 9 nu u A \ (J4 n 9 nu u A ), but Sp(24,C), Sl(6,C)//i 3 and Sp(6,C) do not admit 
irreducible representations of dimension 22. Presumably G = Sp(22,C). 

6. Tensor functors defined by nearby cycles 

To study the tensor categories P(X) or P(X, 0) of section 1331 when (X, 0) 
varies in families, we briefly recall some facts from nearby cycle theory as ex- 
posed in [SGA71 exp. XIII-XIV] and |KSj . Let / : X — > S be a proper surjective 
algebraic morphism from a smooth complex algebraic variety X to a smooth 
complex algebraic curve S. For s G S(C) we want to relate complexes on the 
special fibre X s = f^ 1 {s) to complexes on the nearby fibres X t = for t in 

a small pointed analytic neighborhood of s. 

6.1. Analytic nearby cycles. Let D C S(C) be a small coordinate disc cen- 
tered at s. The morphism / : X — > S induces a proper holomorphic map 
Xd = XxsD^-Doi analytic spaces, ditto with D replaced by the pointed 
disc D* = D\{s} ot by the universal covering D* of D*. So we have a cartesian 
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diagram 

v f 1 v ^ 1 v v 
A s >■ Ifl -< Afl. -< sljj, 

{ S }C £) -< ^ £,* 

where 7r is a covering map and i resp. j are closed resp. open immersions. Let 
j = j o 7i. Following |SGA7[ exp. XIV] and |KS[ sect. 8.6] we consider the 
functor 

iff: Dc(X, C) — > Dc(X s , C), 5 ^ i* Rj*~f (8\ Xd ) 

of nearby cycles. It factors over a functor D b c (X D ,,C) -> C) which 

we also denote by iff. If D has been chosen sufficiently small, one has an 
isomorphism 

(6.1.1) W(X s ,iff(8)) = W(X t ,8\ Xt ) for all teD*. 

For 5 G Dc(Xjj,C) the pullback of the morphism 8 — > j*j*(8) under i defines 
a morphism i*(8) — > iff (8). As in [KS, eq. 8.6.7] the cone of this morphism 
defines the functor $ : Dc(X£,, C) — > D C (X S1 C) of vanishing cycles, so for 
every 8 G Dc(Xd, C) we have a distinguished triangle 

(6.1.2) — ► *(<J) — ► *(<5) — ► 

in Dc(X s ,C). Be aware of the various shifting conventions in the literature; 
e.g. our $ would be denoted 1] in [KSJ. With our conventions, the shifted 
functors \&[— 1] and $[—1] commute with Verdier duality |Br| 1.4], and they 
map Perv(X) to Perv(^) jKSl cor. 10.3.13]. 

6.2. Tensor functoriality. If X — > S is an abelian scheme over S and if 
a iPiiP2 '■ X x s X — > X are the relative addition morphism resp. the two pro- 
jections, we define the relative convolution of Si, 82 G D%(X, C) to be 8% * 82 = 
Ra*(pl(6i) (8)^2(^2)) • For all t G S(C) it restricts on the fibre X t to the con- 
volution (81 * $2)\x t = $i\x t * $2\x t as defined in 13.11 The Kiinneth isomor- 
phism |KSl dual of ex. 11.18] and the compatibility of iff with proper maps \KS\ 
ex. VIII. 15] implies that \1/ is a tensor functor in the sense that for all Si, 82 one 
has iff (Si * 8 2 ) = iff (81) * iff (8 2 ). 

6.3. Algebraic nearby cycles. Localizing in the point s G S(C), let us now 
replace S by the spectrum of a Henselian discrete valuation ring centered at the 
special point s, and denote by 77 its generic point. We then have an algebraic 
version of the nearby cycles, a functor iff : D^(X) — > Dl(X s x s 77) in the sense 
of (III §3.1 and §4]. It factors over a functor D C (X V ) — > Dc(X s x s 77) which we 
also denote by iff. The properties described in the analytic setting carry over to 
this case. We have a sequence (16.1.2)) . and iff maps Perv(X v ) to Perv(X s ) and 
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commutes with Verdier duality by th. 4.2 and cor. 4.5 of loc. cit. Furthermore, 
if X —7- S is an abelian scheme, \I> is a tensor functor with respect to convolution 
on X v resp. X s by compatibility with proper maps |SGA7l exp. XIII, 2.1.7] and 
the Kiinneth formula |BB| 5.1]. 

6.4. Group-theoretical reformulation. For an abelian scheme X — > S, in 
the setting of I6.3[ let T be a tensor subcategory of P m ixed{^n)- Denote by 7i 
be the tensor subcategory of P m ixed(X s ) generated by the image ^(T). If T is 
a finitely generated tensor category, so is 7*. Then there are algebraic fc-groups 
G and Gq, such that T = Rep(G) and 7* = Rep(G^), where the right hand 
sides denote the tensor categories of algebraic representations of G resp. G\$i. 

Lemma. The functor ^ is a k-linear £g! -functor ACU and maps perverse sheaves 
in T(X V ) to perverse sheaves in T(X S ). Hence it induces a k-linear exact <g>- 
functor 

V : Rep(G) — > Rep(G*). 

Proof. The algebraic analog of the isomorphism (I6.1.ip shows that the func- 
tor \l/ maps complexes with vanishing Euler characteristic to complexes with 
vanishing Euler characteristic. So we get a £g>-functor 

P mixed^^r)) ^ P mixc.d\^-n) ; 

which immediately implies the assertions. Notice that $ maps distinguished 
triangles to distinguished triangles, hence induces an exact functor. □ 

6.5. Deligne [Del6, sect. 8] has attached to any Tannaka category T an Ind(T)- 
groupscheme vr(T), called the fundamental group of T. By 8.15 of loc. cit. any 
fc-linear exact ®-functor i] : T\ — > 7~2 induces a morphism 

(6.5.1) 7r(T 2 ) — > v{k{Ti)). 

Under the weak conditions (2.2.1) and (8.1) of loc. cit. (which are verified for 
representation categories % = Rep(Gj) of algebraic groups Gi over an alge- 
braically closed field k of characteristic zero) theorem 8.17 of loc. cit. implies 
that the functor rj induces an equivalence of 7i with the category of objects in 
1~2 endowed with an action of ?7(vr(7i)) such that the natural action of 7r(72) 
is induced by (16.5. ip above. If r] is a fiber functor to the tensor category 
72 = Veck of finite-dimensional vector spaces over k, this reduces to the as- 
sertion 71 = Rep(G\) for G\ = t](tt(Ti)). 

6.6. Let 71 = 7" be a finitely generated tensor subcategory of Pmixed{X-q) as in 
section [63] an d choose a fiber functor u> of T2 = 7i. Then uoty is a fiber functor 
of 7i = T, since it is exact and therefore faithful by the isomorphism (I6.1.ip . 
Hence (I6.5.ip applied to the functor \1/ induces a morphism of algebraic /c-groups 

Gyp = G2 — y G\ — G. 
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In [DMt p. 118] it is shown that this morphism G^ — > G is a closed immersion 
iff every object K of 72 = Rep{G-^) is isomorphic to a subquotient of an object 
ty(K') for some K' in 7i = Rep(G). In our situation this holds by the definition 
of 7$, so we get the 

Lemma. The algebraic k-group Gq, is a closed algebraic k-subgroup of G, and 
\I/ can be identified with the restriction functor Rep(G) — > Rep(G^). 

6.7. In dealing with tensor categories of mixed perverse sheaves on abelian 
varieties X over k one can use the following 

Lemma. Let (5) = Rep(G) be the full tensor subcategory of P m i xe d(X) gener- 
ated by a mixed perverse sheaf 5. Then the full tensor subcategory generated by 
the semisimplification 5 SS of 5 is 

(S ss ) = Rep(G red ) 

where G red = G/R U (G) denotes the quotient of G by its unipotent radical. 

Proof. This is just a statement about the categories of representations of 
algebraic groups over a field of characteristic zero; see |KrW2] . □ 

7. Local Monodromy 

In the setting of 16. 1[ let 5 e Perv(A')[— 1]. Then ^(5) is a perverse sheaf on 
X s , and for fixed t G D* the action of 7Tj = iri(D*,t) on the universal cover D* 
induces a monodromy operation on this perverse sheaf. In the algebraic setting 
of section 16.31 we can proceed similarly, replacing tti by the pro-cyclic local 
monodromy group Zj(l) as in m §3.6], cf. 17.41 below. 

7.1. Unipotent nearby cycles. Let T be a generator of -k\ acting on *&(5) as 
above. We have a direct sum decomposition 

*(<$) = #i(5)©^i(5) 
where ^i(S) C \I/(<5) denotes the maximal perverse subsheaf on which T acts 
unipotently |Reil lemma 1.1]. Similarly = $i(<5) © We say 5 

has unipotent global monodromy if M*(X s ,^i(5)) = M*(X S , ^(5)). From the 
Picard-Lefschetz formulas |SGA7| exp. XV, th. 3.4(iii)] one draws the 

Lemma. // X s is regular except for finitely many ordinary double points and 
if &\m{X s ) is odd, then 5 = 8x[— 1] has unipotent global monodromy; more 
precisely (T — l) 2 acts trivially on M*(X S , v I / (5)). 

Returning to the general case, since by definition T — 1 acts nilpotently 
on *j(<J), we can define N = J- log(T) : — > *i(<J)(-l). The cone of N 
in Dc(X s , C) is given by 

0(^(6) Am^X-I)) = C(*(S)^*(6){-1)) = i*Rj*f(6[l))- 
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Indeed, the first equality holds because T — 1 is an isomorphism on ^f^i(5) 
whereas on ^i(5) its kernel and cokernel coincide with those of iV up to a weight 
shift. For the second equality see [II, eq. (3.6.2)] and the remarks thereafter. 
The perversity of \E r i(5) and the above formula for the cone of N imply that if 
we define specialization functors by 

sp(-) = W°(i*Rj*f(-)) and sp\-) = W\? Rj,j*(-)), 

we obtain an exact sequence of perverse sheaves on X s 

— > sp{5) — > *x(5) A — sp\5) — > 0. 

Since \l/ and hence also preserve distinguished triangles, the functor sp is 
left exact on perverse sheaves. 

7.2. The monodromy filtration on ^ (5). As in [Del3l section 1.6] the nilpo- 
tent operator iV gives rise to a unique finite increasing filtration P. of in 
Perv^s) such that for all i, 

• N(Fi(^i(5))) C F l _ 2 (^ 1 (5))(-1), and 

• N l induces an isomorphism Gri(^i(5)) Gr_j(\E' 1 (5))(— i). 

Each Gr_j(\l/i((5)) with i > has an increasing filtration with composition 
factors pj(6), P_i_ 2 («J)(-l), P_i_ 4 (5)(-2), . . . where 

Pi{6) := ker(iV : Gr^S)) -> Gr i _ 2 (^ 1 (5))(-1)), 

In what follows we will represent this situation as in loc. cit. by a triangle 



Gr^tf)) P-2{5){-2) 

Gr x ^ x {5)) P-i(S)(~l) 

Gr„(tfi(<5)) P (5) P_ 2 (<5)(-1) 

G?r_i(*i(5)) P_ a («J) *Jiv 

Gr.aC*!^)) P_ 2 (5) 



where each line gives the decomposition of the corresponding graded piece. The 
lower boundary entries Po(5), P_i(5), P_ 2 (5), ... in the triangle are the graded 
pieces of sp(5) = ker(iV), with Pq(5) as the top quotient. In the situation of 
proposition 18.1( a) below, the entries above these lower boundary entries belong 
to 
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7.3. The monodromy filtration on ^(5). For any 5 G Peiv(X)[— 1], the 
local monodromy theorem |TT|, th. 2.1.2] and the Jordan decomposition of the 
nearby cycles |Reit lemma 4.2] show that there is an a G N such that T a — 1 is 
nilpotent on all of $f(5). Using N' := ^ log(T a ) : #(8) ->■ $f(5)(-l) in place 
of N : ty\(5) — > \E'i((5)(— 1) one can then define a filtration F' 9 as in 17.21 on all of 
ty(S). This filtration does not depend on the choice of the integer a with T a — 1 
nilpotent. Even though in general N'\^ 1 (s) 7^ N, using that T acts unipotently 
on ^i(8) one sees that the kernel and the image of N'l^^s) are the same as 
those of N, so 

F.^S)) = $!((5)nF:(^)). 

Notice however that sp(5) = ker(iV : *&i(8) — > \E f i(5)(— 1)), as defined in l7.lt will 
in general only be a perverse sub sheaf of ker(N' : — > ^f(8)(— 1)) because 
iV may have a non-trivial kernel on On the other hand, working with 

^(5) instead of ^\(5) has the following advantage. 

7.4. Tensor functoriality. All of the above has an analog in the algebraic 
setting of section 16. 'S\ if T is a topological generator of the local monodromy 
group Z/(l) as in p]] §3.6]. 

Lemma. For 5 G Pervf^), denote by Gr' 9 (^(5)) = ieZ Gr' i ('9(8)) £ne asso- 
ciated graded with respect to the filtration F' t on ^(8) as defined above. Then 
we have an induced functor 

Gr'.oV: P(X V ) _> P(X S ), 8^Gr',^(8)) 

which is a tensor functor with respect to convolution. 

Proof. By lemma I6.4[ \1/ induces a tensor functor P{X r) ) — > P m ixed{X s ) 
with respect to convolution. Furthermore, if a fixed generator of Z^(l) acts 
on 5i G Perv(A^) via endomorphisms Tj : ^(8i) — > \I/(<5j) for i G {1,2}, then 
this generator acts on \l/(o*i * 5 2 ) = ^(^l) * ^(82) via T\ * T2. Since the tensor 
subcategory of P m ixed{Xs) generated by \l/(<5i) and \J/(<5 2 ) is equivalent to the 
category of representations of some algebraic group, as in |Del3t prop. 1.6.9] 
one deduces Gr'0^) * *(6 2 )) = ® il+h=i G^(*(*i)) * Gr' h (*(8 2 )). □ 

7.5. Weights. Concerning weights we have the following observation due to 
Gabber [Sa2l 1.19] [BBj th 5.1.2]. 

Lemma. If 5 underlies a pure Hodge module of weight w on X v , then each 
graded piece Gr'^^)) is pure of weight w +i and the filtration F^(^(8)) is the 
weight filtration of ^(8) up to an index shift. Hence the analogous statement 
also holds for the filtration F, of ^ 1(8) . 
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8. Behavior of G(X, 0) under specialization 

In this section we study the behavior of the Tannaka group G(X, 0) when the 
ppav (X, 0) degenerates. For this we need to control the specialization functor 
sp as defined in 17.11 

8.1. Alternative description of sp(S). Let 8 E Perv(X)[— 1]. Then with 
notations as in 16.11 we claim that 

sp(8) = 1] for the perverse sheaf 7 := 5[1]|^ D ». 

Indeed, a basic property of intermediate extensions |KWl 111.5.1(7)] shows that 
i*3\*l = p T<oi*Rj*1, and one has p r <0 i*_Rj^7 = sp(5)[l] since j is affine so that 
i*Rj*j E PD [-1,OI (A!b) by Artin's vanishing theorem [BBDl th. 4.1.1]. See also 
the formula for the cone of N in section 17.11 

Proposition. From the formula sp(5) = i*{j\*^f)[— 1] one draws the following 
two basic observations. 

(a) If 5 = (j'ut)[-1] = J!*(5[1]U D «)[-1L then sp(5) = i*(8), so (KT^) yields 
an exact sequence of perverse sheaves 

— ► sp(5) — > — > $(5) — ► 0. 

The same holds with \l/ and $ replaced by and $1 since T acts trivially 
on sp(8). 

(b) If 5 underlies a Hodge module of weights < w, so does sp(5). This 
follows from the permanence properties of weights under pull-back and 
intermediate extensions. 

8.2. In situation (a), the local invariant cycle theorem |BBDl cor. 6.2.9] states 
that for all % e Z the induced morphisms M l (X s , sp(5)) — > M l (X s , ^(5)) factor 
through epimorphisms 

W(X s ,sp(8)) -» W(X S1 V(5)) T 

onto the invariants under the local monodromy group. In (b), if 5 is pure of 
some weight w, we denote by ~sp(8) the highest top quotient (of weight w) of 
the weight filtration of sp(8). 

Example, (i) If X is smooth over S of relative dimension d, then 

sp(£ x [d]) = C Xs [d}- 

(ii) If X is regular of dimension d + l, then sp(Cx[d]) = C Xs [d\. If in addition 
X s is normal and if the only singularities of X s are finitely many ordinary double 
points, then sp(Cx[d]) = 8x 8 - 

Proof. Part (i) is obvious. To check the first statement in (ii), note that the 
regularity of X implies j\*j*{Cx[d+ 1]) = Cx[d + 1]. For the second statement 
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in (ii) see lemma I12.1[ noting that in the case at hand X s is automatically a 
local complete intersection. □ 

8.3. As an immediate corollary, part (i) of the above example applied to a 
smooth family of theta divisors gives the following 

Rigidity lemma. If conjecture^ holds for a single ppav in A g \ M g , then it 
holds for all ppav's in A g \ M g . 

Proof. Any two ppav's in A g \ Af g with corresponding groups G%, G 2 can be 
connected by a sequence of smooth curves in A g \M g . These curves give smooth 
families of ppav's whose relative theta divisors are smooth over the respective 
base curves. From part (i) of the example and section R3I3I we conclude G\ <-> G%- 
Hence by symmetry G\ = Gi- D 

8.4. Now, for a principally polarized abelian scheme (X, Q%) over a discrete 
valuation ring S as in sections 16.31 and 16.4} we write Q s = Qx s an d Q v — ®x v - 
Then we have the following 

Lemma. If Qx and @s are normal, then ~sp(5@ v ) contains 5 S . In particular, 
then G s = G(X S ,Q S ) is a subquotient of G = G(X v ,Q rj ), and the defining 
representations of these groups satisfy dim^cu^ej) < dim.k(oj(5e )) . 

Proof, a) If 5e g is a constituent of ~s~p(5q ), then G s is a quotient of the 
Tannaka group Gy, and lemma [6761 implies ^ G. So the second statement 
of the lemma follows from the first one. It remains to show that, if 0^ and O s 
are normal, 5q s is a constituent of sp(5e ). 

b) If Qx is normal, lemma [T2.1( ii) gives an exact sequence of perverse sheaves 
— > ipe x — » k@ x [g] — > 5e x — > where i/jq x has weights < g. Recall that 
i*SQ x [— 1] = ^^(^e^) is perverse by proposition 18.11 Hence, if we apply 1] 
to this sequence, we get an exact sequence of perverse sheaves on X s 

PH°(i*Tp ex [-l)) k Qs [g - 1] -> sp(5e v ) p H\^ ex [-l]) , 

since G s is a local complete intersection and therefore also i*k@ x [g — 1] = 
ke g [g — 1] is perverse. Since the restriction functor i* preserves upper bounds 
on weights, the first term p H°(i*i(}Q x [—l\) has weights < g — 1. 

c) Lemma 112.1( 1) applied to the special fiber gives an exact sequence of per- 
verse sheaves 

-> ipe s -> k @s [g - 1] -> 5© s ->■ 
where ^e s has weights < g> — 1 with pure quotient 5q s of weight g — 1. Thus the 
perverse sheaf fcejfl 1 — 1] admits 5& s as the highest weight quotient of weight 
g — 1. By weight reasons and the exact sequence of perverse sheaves in b), 
this epimorphism factorizes over the quotient perverse sheaf sp(5& ), because 



22 



T. KRAMER AND R. WEISSAUER 



P #°(i*</W-1]) has weights < g — 1. Again by weight reasons the epimorphism 
then also factorizes over ~sp(5@ v ), i.e. we get an epimorphism ~sp(5@ v ) -» <5e s . □ 

In passing we remark that, if in the setting of lemma 18.41 the assumption 
that S is normal is replaced by the assumption that 0.,, is smooth, then the 
argument in b) does imply that kQ s [g — 1] is a perverse subsheaf of sp(5q ). 
Indeed, then ip® x = i*(a) for some perverse sheaf a supported in the special 
fiber Q s , so that p H°(i*il)Q x [— 1]) vanishes since i*ipQ x = a is perverse. 

9. Proof of conjecture [B] for g = 4: Outline 

To prove conjectures [B] and [UJ let (X, 0) be a ppav of genus 4 with smooth 
theta divisor O. By the rigidity lemma in section and by section [2] we may 
assume MT(0) sc = MT(X) x Sp(10,C). We have to show that 5± are simple 
objects in the representation category Rep(G(X, 0)) = P(X, 0). For this we 
compare two tensor functors, 

(a) the global motivic functor MT related to the Mumford-Tate group of 0, 
and 

(b) a restriction functor induced from an embedding ^ G — G(X, 0) 
of algebraic groups via the theory of vanishing cycles, obtained by a 
degeneration of X into the Jacobian JC of a general curve C of genus 4. 

To describe the first functor recall the realization functor from the abelian cat- 
egory MHM(X) of mixed Hodge modules to the abelian category Perv(X) of 
perverse sheaves, and similarly the functor D fe (MHM(X)) — > Dc(X,k). We 
consider the tensor subcategory MHM(X, 0) C D b (MHM(X)) over P(X, 0) C 
D b c {X,k). The direct image under the structure morphism X — > Spec(C) in- 
duces a tensor functor to the category of fc-linear finite dimensional super rep- 
resentations of the group MT(X, 0), 

MT:MHM(X,0) — ► sRep(MT(X, 0)). 

To define the second functor we let (X, 0) degenerate into the Jacobian JC 
of a general curve of genus 4. For the details of the construction we refer to 
section 110.11 We show that the subgroup G^ C G = G(X, 0) defined by this 
degeneration contains the group G(JC,Qjc) = Sl(6, C)//z 3 (see |Wel| ). 

We thus obtain a diagram of ®-functors, where the vertical £g>-functor is the 
composition of the realization functor MHM(X, 0) — > P(X, 0) and the quotient 
functor P(X, 0) -> P(X, 0): 

MHM(X, 0) sRep(MT(X, 0)) sRep(MT( JC, JC )) 



P(X,Q) 



Rep(G) 



Rep(G(JC,0 JC )) 
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Notice that the thereby defined functors MHX(X,Q) -»■ sRep(MT(X, 0)) 
and MHX(X, 0) — > Rep (G(JC, 0jc)) are completely unrelated to each other. 
Since unlike the second functor the first functor is non-trivial on T(X), in order 
to compare the two functors, we have to carefully keep track of all constituents 
from T(X). These constituents arise in the tensor square of the generator 
5 = 5& of the tensor category MHM(X, 0), see lemma 15751 The most important 
step for the proof, that the two summands 8± of the tensor square of 5 are 
simple, is the key lemma [10.51 where we construct two large simple subobjects 
7-t C (5 ± . For the proof of this lemma we compare the decomposition of the ob- 
jects 5± G MHM(X, 0) in the two representation categories sRep(MT(0)) and 
Rep(G( JC, Qjc))- In the decomposition of the image of 8± in sRep(MT(X, 0)) 
there arise many irreducible summands; the complete list can be read off from 
table [TJ in section [T3J On the other hand, by |Wel] the decomposition of the 
image of 8± in Rep(G( JC, Qjc)) has only few summands. In order to compare 
these two apparently unrelated decompositions one needs to know the behavior 
of the Mumford-Tate group for (X, 0) degenerating into (JC,Qjc)- Indeed, 
since the simply connected covering of the Mumford-Tate group MT(0jc) for 
a general curve of genus 4 is isomorphic to the group Sp(8, C), this behavior is 
decribed by a group homomorphism 

MT (JC, &j C )sc = Sp(8, C) -»■ Sp(8, C) x Sp(10, C) -»■ MT(X, 0) 

where the first homomorphism is the diagonal embedding (id, if) defined in I1U.4I 
and where the second homomorphism is the one from section [2j Finally, in the 
last step of the proof we also need to consider the action of the involution o on 
the perverse sheaves involved, and their weights, in order to control the many 
summands arising from table [TJ Using this we show in subsection 110.61 that 
7± = 8±. 

10. Proof of conjecture [B] for g = 4: The details 

In order to apply the results of section [2J we give the proof in an analytic 
framework. For this recall that if V is a complex algebraic variety, Dc(V, k) is a 
full subcategory of the derived category of complexes on the analytic space V an 
which are constructible for an algebraic stratification [BBD, 6.1.2]. Hence it is a 
full subcategory of the category of all C-constructible complexes on V an in the 
sense of [KSJ. The number of constituents of a semisimple complex is the same 
in any of the above triangulated categories in which it lies, so we will denote all 
of them indifferently by D^(—, C). 

10.1. Degeneration into a Jacobian. We first construct the degenerating 
family of ppav's to be used in the proof. Let C be a generic smooth algebraic 
curve of genus g — 4 over C and (JC, Qjc) its Jacobian variety. Recall from 
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lemma [57T1 that the theta divisor Qjc has precisely two distinct ordinary double 
points ±e as singularities. 

Lemma. There exists a principally polarized abelian scheme (X, ®x) over a 
smooth quasi-projective curve S , 

G x < X 

S 

and a point s G S(C) such that 

• the fibre in s is (X s , O s ) = (JC, Qjc), 

• the total space X and the relative theta divisor x are nonsingular, 

• over S* = S \{s}, the structure morphisms Xs* = X x 5 S* — > S* and 
Qx* = ®x Xs S* — > S* are smooth. 

Proof. Write JC = C 4 / (Z 4 + T0Z 4 ) for some point To in the Siegel upper 
half plane H 4 , and choose lifts ±z in C 4 of ±e G JC. Let it : H4 — » A<± 
be the analytic quotient map. Since ±e are ordinary double points of Qjc, 
by the heat equation the gradient of the Riemann theta function 8(t, z) in r- 
direction does not vanish at (r , ±^o)- Since is quasi-projective, we can use a 
suitable system of regular parameters of the regular local ring at tq to construct 
Zariski-locally a smooth algebraic curve S C A4 intersecting J/4 transversely 
in s = tt{tq) in a tangent direction along which the gradient of r 1— > 6(t, ±Zq) 
does not vanish at To. We define (X,Qx) as the restriction of the universal 
ppav (O x U g )l{l? g x Sp(8, Z)) -> A 4 to S, shrinking S if required. □ 

10.2. Some notations. To transfer the constructions of 13.41 to the relative 
situation 110.11 consider the pure Hodge module 5 = Sq x [— 1] = Ce^[3]. By 
Gabber's theorem its relative symmetric resp. alternating convolution square 
decomposes as S 2 (S) = 5 + © r + resp. A 2 (5) = 1% © 5- © T- where \x is a 
complex supported on the zero section of X and concentrated in degree zero, 
where 5± G D^.{X, C) are semisimple complexes and where as in lemma 13.51 
one has r± = 0^ a±(mod2) t (R-^ U(5 x )) © 8 X [^ - 1]. For each * G S*(C) 
the restrictions r-t]^ and 8±\x t are the complexes that were previously denoted 
by t± resp. 5±. To prove conjecture [Bl it clearly suffices to show that the new 
5± G Dc(X, C) are simple. 

We consider equivariant perverse sheaves |KW[ section III. 15] with respect 
to the involution a = —idx B . Let l a ± be the cr-equivariant skyscraper sheaf 1 
with a acting by ±1. For x G X s \ {0} we denote by l± x = t*(l) © t*_ x (l) the 
simple cr-equivariant skyscraper sheaf supported in {±x}, with a flipping the 
two summands. 
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10.3. Monodromy filtrations. Recall that by construction Q s = ®x B is reg- 
ular except for two distinct ordinary double points ±e. In particular, then 
5 = Sq x [—1] has unipotent global monodromy by lemma [T7T1 From section IT721 
we deduce that the monodromy filtration diagram of is 



l± e (-2) 

N 



L ±el 



since sp(S) = Xe s by ex. 18.41 and since the weight filtration of Ae s is defined by 
the exact sequence — > l± e (— 1) — >■ Ao s — > 5q s — > of lemma [12.11 

Remark. The above implies x{$e s ) = xO^i(^)) ~ 4 = g\ — 4 = 20. More gener- 
ally, for any g > 4 and r G N, similar arguments show that for a ppav (X s , Q s ) 
whose theta divisor O s has precisely r ordinary double points as singularities, 
one has x{$e s ) = gl — 2r. 

Now consider Wi(6±). Clearly S 2 (l± e ) = l± 2e © 1 CT + and A 2 (l ±e ) = l a . 
Furthermore, with notations as in |Wel] . theorem 14 on p. 123 in loc. cit. and 
the representation theory of SI (6, C) imply that 

S 2 (6 &a ) = 5 3 ,3©<*5,i and A 2 (5 J = <5 4 , 2 © <J fl)0 . 

In what follows, we write 5 a = p 5 a © c 5 a where c 5 a is a direct sum of complex 
shifts of Sx s and p 5 a has no constituents in T(X). Here c and p stand for the 
properties of being constant resp. perverse. 

Lemma. Up to constituents with vanishing hypercohomology, for \E' 1 (5 + ) the 
monodromy filtration diagram is 



;i±2e©l«T+)(-4) 



(5 Qs * l ±e )(-2) 



P 53,3© P 5 5 ,l©l CT -(-3) 



N 



N 



;i±2e©l CT+ )(-3) 



(6 Qs * l±e)(-l) 



N 



;i ±2e ® i CT+ )(-2) 
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with graded pieces of weights 4, 5, 6, 7 and 8. For one has the diagram 

l.-(-4) 



(fe.*l±e)(-2) 



^4,2©(l±2e©l CT+ )(-3) — N l«r-(-3) 



IV 



(5e.*l±e)(-l) 



A' 



1.4-2) 



witt graded pieces of the same weights as above. 



Proof. Use lemma 17741 and the fact that \l/i(5±) differs from \&(5±) at most 
by constituents with vanishing hypercohomology, taking into account that 5± 
has unipotent global monodromy. To lift the obtained result from P(X S ) to 
Perv(A' s ), note that the complex translates of 5x Q which enter *f?i(S 2 (5)) and 
\&i(A 2 (5)) can be computed as in 13. 5[ so no additional such terms occur in 
*i(&t). ' □ 

10.4. The degenerate Hodge structure. For a suitable choice of (X,Q X ) 
in lemma [10.11 and suitable (fixed) t G S*(C) we abbreviate Q t = ®x t - Then 
section [2] shows 

MT(Q t ) sc = MT(X t ) x Sp(10,C) where MT(X t ) = Sp(8,C). 

In section [T31 we compute the natural pure Hodge structure on W(Xt, 8±\x t ) 
as a representation of this group MT(Q t ) sc . But W(Xt, <5±|;f t ) can a l so be 
equipped with a different (mixed) Hodge structure, induced under the isomor- 
phism W(Xt, 5±|^,) — M*(X S , ty(5±)) from the Hodge structure on the hyper- 
cohomology of the mixed Hodge module ^(S±). 

Let us call this Hodge structure on W(Xt, $±\x t ) the degenerate Hodge struc- 
ture. Its subquotient Hodge structures — e.g. its invariants under the mon- 
odromy operator N — are no longer representations of MT(Gi) sc , but they are 
representations of MT(9 S ) SC = MT(C) = Sp(8,C). 

To obtain the degenerate Hodge structure on W(Xt, 5±|^,), one easily sees 
that in table [1] of section [13] the representations of MT(X t ) = Sp(8, C) are un- 
changed (including their weights) when viewed as representations of MT(^Y S ) = 
Sp(8,C). In particular, all of them are invariant under the monodromy opera- 
tor N. However, for the standard representation B of Sp(10, C) the situation 
is different as we will see in the lemma below: The degenerate Hodge structure 
arises from the natural one by pull-back along a monomorphism 

(id,<p) : MT(6 S ) SC = Sp(8,C) ^ MT(6 4 ) SC = Sp(8,C) x Sp(10,C) 
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for some embedding ip : Sp(8, C) Sp(10, C). 

Put A* = H l (X s , Q) for % G N. As representations of the group Sp(8, C) these 
A* are considered as the exterior powers of the standard representation. Notice 
A = (0000), A 1 = (1000), A 2 = (0100) © A and A 3 = (0010) © A 1 in the 
notations of section [TBI 

Lemma. On the quotient B of H 3 (Q t ,C) = H°(# t ,5 et ) - B?(X 8 ,V(6)) with 
its degenerate limit Hodge structure, the monodromy operator N has the coin- 
variants 

B/B N = A°(-2) 

which are pure of weight 4, and the weight filtration of the invariants B N is 
given by an exact sequence 

— ► A°(-l) — ► B N — >• A x (-1) — ► 

A x (— 1) pnre o/ weight 3 and mt/i A°(— 1) pnre o/ weight 2. 

Proof. Since $(5) = l± e (— 2) and sp(5) = Ae s by section we get from 
proposition 18.1( a) an exact sequence 

H°(^, A J M°(Ar t> <J 0t ) -A H°(/fo, l± e (— 2)) 

=A 3 ©_B =(0000)® 2 

where a and /3 are morphisms of Hodge structures if the middle term is equipped 
with the degenerate limit Hodge structure. By the local invariant cycle theorem 
the image of a is the subspace of AMnvariant elements. Since dime (B) = 10 
and dimc(A°) + dim^A 1 ) = 9, by a dimension count the claim follows once we 
can exhibit an exact sequence 

(10.4.1) — ► A°(-l) — ► U°{X S , A©J — + A 3 © A\-l) — >• 
The sequence (110.4. ip is obtained as the short exact sequence 

(10.4.2) — > ker(H») — > H°(^, A e J — * BT°(^ fl , <J e .) — )• 

obtained by splicing up the long exact sequence attached to the short exact 
sequence -> l ±e (— 1) -> A 0S -> #e s -> of lemma [12T1] i.e. with ker(H°(zJ) 
isomorphic to the quotient 

(10.4.3) — ► M~ 1 (X S , A e J — > M~ 1 (X S , 5 @s ) — > ker(H°(z/)) 0. 

The right hand side of (110. 4. 2p and the middle term of (110.4.31) are given by 
the cohomology of the smooth curve C via H , (5e s ) = A 9 ^ 1 (M.'(5c)), see |Wel| . 
Thus 

E°(X s ,Sq s ) = A 3 ffiA 1 (-l) and U-\X S ,5 @S ) = A 2 ©A°(-1). 

To finish the proof it remains to compute the left hand side H _1 (Af s , AeJ 
of ( I10.4.3p . For this use the long exact cohomology sequence associated with 
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the triple (X s \ Q s , X s , S ). Since H l c (X s \ Q s , C) = for i < 3 by the ampleness 
of S C X s , this implies 

M- 2 (X s ,X Xs ) = H 2 (X S ,C) ^ H 2 (Q S ,C) = H- 1 (Af„A e .), 
which is isomorphic to A 2 . From this one easily concludes the proof. □ 

10.5. The simple constituents j± of 6±. We now exhibit two irreducible 
constituents j± 5± such that sp(7±) differs from sp(5±) at most by skyscraper 
sheaves. For wq 6 Z and a mixed perverse sheaf 7r on X s , we denote the maximal 
perverse subsheaf of 7r of weights < Wo by 7i w<Wiy 

Key lemma. There are unique irreducible constituents j± <^-> <5± m Perv(A^) 
wi/i p 55 5 i © ^3,3 <^-> sp(7+) and P S^2 ^ sp(7_) zn Perv(A' s ). T7ie perverse 
sheaves sp(7+) w <6 and sp(7_)„,<6 &ot/i admit 5e s * l± e (— 1) as a quotient. 

Proof. Recall from lemma [10.31 that K = 5@ s * l± e (— 1) is a a-equivariant 
simple constituent of sp(5±) w< Q. Suppose e± is a constituent of 5± for which 
sp(e±) does not contain K. Then sp(e±) w< 6 and $(e±) are skyscraper sheaves, 
hence 

M- 3 (X s ,^ 1 (e ± )) = and U~ 2 (X S , sp(e±)) ^ W~ 2 (X S , *i(e±)). 

By the local invariant cycle theorem we then have an isomorphism 

H- 2 (* S , ^> M- 2 (X t ,e ± \ Xt ) N . 

To compute H _2 (Aft, £±1^)^ we use the second line of table [1] in section [TBI 
where H _2 (A' i , 5±|^J is listed. The monodromy operator iV acts non-trivially 
only on B, and B/B N and B N were computed in lemma [10.41 

We claim that the summand (lOOO)®^ of H" 2 ^, 5±\ Xt ) N is linearly disjoint 
from M~ 2 (X t , e±\x t ) . Otherwise the summand (1000) (&B N would be contained 
in W~ 2 (X S} ~sp(e±)) by global monodromy reasons since e± are complexes defined 
globally on X. Namely, (1000) ® B would occur in W(X S , *(e±)) = M'(^,e±) 
as a representation of the Mumford-Tate group, so by the local invariant cycle 
theorem (1000) <S> B N would occur in M~ 2 (X S , ~sp(e±)). This is impossible, since 
B N is not pure. 

By section [TBI our claim shows HT 2 (;f s , sp(e±)) C (1010) © (0100). Hence 
again by section [131 t ne simple perverse sheaves 

L = p 05,i, p 54,2 or ^3,3 

cannot be constituents of ~sp(e±), since for these L the representation of MT(X S ) = 
Sp(8, C) on B.- 2 (X S , L) is not contained in (1010) © (0100) (see line 2 of table EJ). 

Hence for any cr-equivariant constituent j± of 5± such that sp(7±) contains 
one of the constituents L above, sp(j±) has K as a constituent. Since from 
lemma [10.31 we know K enters with multiplicity one in sp(S±), it follows that 
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there are unique a-equivariant simple constituents j± of 5± containing one of 
the simple perverse sheaves L above. These satisfy 

p 5 5 ,i© p <53,3 ^ sp(l+) and p 5 4 ,2 sp(l-), 
which easily concludes the proof. □ 

10.6. Excluding skyscraper sheaves. Now define e± G D^(X , C) by the de- 
composition 5± = 7±ffi£± °f semisimple perverse sheaves. To prove conjecture |B1 
we must show e± — 0. From lemmas 110.51 and 110.31 we know 

sp w<6 (e + ) <—t (l ±2e © l CT+ )(-2), sp{e + ) m> 1 ct -(-3), 

In particular, sp(e±) and hence ^f(e±) are skyscraper sheaves, so W > (X s ,e±) is 
a direct sum of trivial representations (0000). A look at table [1] shows that 
these can only arise from lines 2 and 4 of this table; possible candidates arising 
from (1000) (g> B in line 2 are ruled out by global monodromy reasons since B 
is an irreducible representation of MT(O t ). Hence the irreducibility of l± 2e as 
a a-equivariant perverse sheaf implies 

sp{e + ) = sp w<6 (e + ) e — >■ l CT +(-2) and sp(e:_) = sp(e_) ^ l CT+ (-3). 

We now claim sp(e: + ) = 0. Indeed, otherwise M?(X a , sp(e)) would be the trivial 
representation (0000) of weight 4. But the trivial representation could only 
arise from one of the summands 

(S 2 (B)) N , (0010) ®B N or (0000) 

in line 4 of the first column of table [TJ The first two of these summands cannot 
contribute because of global monodromy reasons, and the last one has the wrong 
weight 6. This proves our claim that sp(e + ) = 0. But then also \l/(e+) = and 
hence 

W(X u e + \ Xt ) = W(X„ *(e+)) = 
which easily implies e + = since £+\x t 4- T(X t ). 

By the same argument, to show e_ = it suffices to see that sp(e^) = 0. If 
sp(e^) 7^ 0, then by what we have seen above 

sp(e-) = sp(e-) = l ff +(-3), 

hence also ^(s-) = l CT +(— 3) and therefore BP^A^, £-|;eJ is C for z/ = and zero 
otherwise. Therefore x(e_|^) = 1, and by |KrW2| then £-\x t = 1- But this 
is impossible since #e t * 5e t contains the unit object 1 only with multiplicity 
one. □ 
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11. Equivalence of conjectures [B] and \C\ 



In this section we supply the proof of our earlier statement that for any g 
conjectures |B] and O are equivalent. Again let k = Q z or k = C. Let G be a 
reductive group over k and 77 ■=-)■ G a closed subgroup of finite index. For a 
representation {7 of G, denote by Rfj(U) the restriction of U to 77. Similarly, 
for a representation V of 77, let be the induced representation of G. One 
then easily proves the following version of Mackey's lemma. 

Lemma. For every irreducible representation U of G there is a subgroup H' C 
G containing 77 and an irreducible representation V of H' such that the re- 
striction (V) is isotypic and U = I^,(V). 

Corollary. If U is an irreducible representation of G and R^ (U) contains the 
trivial representation, then all constituents of Rq Q (U) are trivial. 

Using this we now prove the equivalence of conjectures [B] and O for all g. We 
must see that IB1 implies ICl If[B]holds, the irreducible representations U = oj(Sq) 
and W± = u(5±) of G = G{X, 0) satisfy 



Then U cannot be a representation induced from a proper subgroup of G because 
otherwise S 2 (U) would contain at least two non-trivial irreducible constituents. 
Hence by the isotypic case of the lemma, Rq (U) = Uq 71 for some irreducible 
super representation U$ of G° and some n G N. 

If n > 1, then for g even resp. odd, Rq (W + ) resp. R^iWJ) contains Uq®Uq. 
Since by self-duality of Uq the trivial representation enters Uo®Uq and since W + 
and W- are irreducible, the corollary would then imply that Uq were trivial. By 
faithfulness then G° would be trivial, i.e. G were a finite group. Then we could 
find K G Perv(X) with uj(K) being the regular representation R, — k[G}. Since 
R® R = R m for m = \G \, then K*K = K® m modulo T(X), so W(X, K) = 
for all i 7^ by |Welt lemma 5, p. 17], a contradiction since 5@ is a direct 
summand of K (every representation of G enters the regular representation R). 

Hence n = 1 and R^ (U) = Uq is irreducible. Then 77^ (^7) contains a 
unique highest weight vector up to scalars, so the same holds for 77^ (5' 2 (^7))- 
Thus 77^o (W+) is neither induced nor isotypic in a non-trivial way, hence by 
the lemma it must be irreducible. It follows that W + is irreducible as a super 
representation of the super Lie algebra q of G. The classification in |KrWl] and 
the fact that dim(V) = g\ then imply that 

(a) if g is even, G is of type or of type C g \/2, 

(b) if g is odd, G is of type D g \/ 2 , 
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and that in all these cases V is the standard representation. On the other 
hand, we have already observed in section 14.31 that the faithful action of G 
on V preserves a nondegenerate alternating resp. symmetric bilinear form /3 : 
V x V — > k, defining an embedding of G into Sp(V, 0) resp. 0(V, (3), for g even 
resp. odd. So G = Sp(V,/3) in case (a). In case (b) either G = 0(V, (3) or 
G = SO(V, 0); but the arguments of |KrW2| show that G does not admit any 
non-trivial character, so G = SO(V, 0) and we are done. □ 

12. Appendix: Two lemmas on perverse sheaves 

Let k = Qi or k = C, and for a variety Y over an algebraically closed field of 
characteristic zero, let ky G D^iY, k) denote the constant sheaf on Y. 

12.1. Weight filtrations. For the computation of nearby cycles we need the 
following refinement of [WelJ lemma 2]. 

Lemma, (i) If Y is an irreducible normal local complete intersection of dimen- 
sion d with singular locus X C Y , then on Y we have an exact sequence of 
perverse sheaves 

— > ip Y — ► k Y [d] — > 5 Y — > 
where ipy is a mixed perverse sheaf of weights < d whose support is contained 
in X and 5y = lCy[d] is a pure perverse sheaf of weight d. 

(ii) If furthermore the only singularities ofY are finitely many ordinary dou- 
ble points yi,...,y n , then ipy = ®2=i^Vi(~^Y i ) ^ s a direct sum of skyscraper 
sheaves for d odd and ipy = otherwise. 



Proof, (i) This follows from lemma 1 and 2 in |Wel] . For the convenience of 



the reader we sketch the proof. Suppose Y is irreducible and normal. Then as 
in loc. cit. there exists a natural morphism v : Xy — > 5y of sheaf complexes, 
such that l-i~ d {v) is an isomorphism. For Y normal Ay = hence Ax 

is a complex of weights < d whereas 5y = lGy[d] is pure of weight d. Since 
Ay G P D-°(Y) by definition and 5y is a perverse sheaf, v factorizes over the 
truncation morphism Ay — > p H°(Xy). Since 8 Y is an irreducible perverse sheaf 
and v is nontrivial, it is easy to see that the induced morphism /i : p H°(\y) — > 
5y is nontrivial. Hence \l defines an epimorphism in the category of perverse 
sheaves. So we obtain a distinguished triangle 

— > ipy — ► k Y [d) — ► 5 Y — > . 

with tpy G P D-°(Y). The long exact sequence of cohomology sheaves for this 
distinguished triangle implies T-L~ u (tpy) = 'H~ u ~ 1 (5y) for v > —d + 2 and 
T-L~ u (ipy) = otherwise, and hence ipy is of weights < d. Finally, if Y is 
also a local complete intersection, it follows from |KW[ III. 6. 5] that Ay itself is 
a perverse sheaf. 
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(ii) Let 7i : Y — > Y be the blow-up of Y in E = {yi, . . . , y n }. Notice 
that Y is smooth. Since tt restricts to an isomorphism over U = Y \ E, by 
purity Rtt*(\ y ) = 5y © 7 for some 7 G Dc(Y,k) with Supp{^) C E, say 
7 = ©™=i © je z(MC? - rf)/2)[j]) ffim - with m 4j G N . Since the ^ are ordi- 
nary double points, the Qi = 7T _1 (t/j) are smooth quadrics of dimension d — 1, 
so by |SGA7l exp. XII, th. 3.3] 

ffc(-f) for je{0,2,...,2(d-l)}\{d-l}, 
H j (Qi,k) = I (k(-t)) 2S for j = d-l, 
[ otherwise, 

where 5 = 1 for d odd and 5 = for 0? even. Now 'H'(R7i*(\ Y ))y i = H'(Qi, k)[d]. 
In particular, 'H d {'i) Vi = and 'H d ~ 2 (7) 3/i = fc, so m^^ = and m i)( i-2 = 1- 
Then = and > 1 for j = d — 2, d — 4, . . . , 4 — d, 2 — d by the hard 

Lefschetz theorem. Another look at stalk cohomology then shows that = 1 
for j = d — 2, d — 4, . . . , 4 — d, 2 — d and = otherwise. Thus 

{fc for r = — d and all y, 

(M-^)) 5 for r = -1 and y G E, 
otherwise, 

with 5 as above. On the other hand, k Y [d] = X Y = H~ d (5Y) = s V<_ d (5y) by 
normality of Y IWeTl lemma 1] and |KW1 III.5.14]. □ 

12.2. Counting IC-constituents. Let Y be a variety of dimension g over an 
algebraically closed field of characteristic zero, and let 5 G Perv(F). 

Lemma. The perverse sheaf 5 admits 5y as a constituent iffM~ 9 (Y,5) 7^ 0. 

Proof. We know 5 G st D-~ g (Y,C), so E%' q = H p (Y,n q (5)) W +<! (Y,5) 
implies M- 9 (Y,5) = H°(Y,V.- 9 {5)). Let j : U <-> Y be open dense such 
that Q = 5[— g]\u is locally constant, possibly zero. Then |KWj III. 5. 14] gives 
H-b(8) = j^Q), hence H (Y,U- 9 (5)) = H (U,Q), and this group is zero iff Q 
has no constant subsheaf. □ 



13. Appendix: Hypercohomology computations 

In this appendix we determine the hypercohomology of some perverse sheaves 
required in section [10J Let (X, 0) be a general ppav in A4 as in section [21 and 
define 5± G Perv(X) as in 13.41 Let JC be the Jacobian of a general curve 
C of genus g = 4, and consider the associated perverse sheaves p 5 a on JC as 
in section 110.31 Let us group the hypercohomology of perverse sheaves into 
packages [n] t which are stable under the Lefschetz operator and occur precisely 
in degrees n, n — 2, . . . , 2 — n, — n for some n G Nq. Denote by (ai, 02, 03, 04) the 
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• 


W(X,5+) 


H*(X,<5_) 


[3]* 


(0000) CT+ ® £ 
(0010) CT _ 


(0000),,+ <g> S 


[2] t 


(1000) CT _ ® J5 

(1010) ff + 

(0100) CT + 


(1000)cr_ (8 B 
(0100) CT + 


[l]t 


(0100) CT+ ® s 

(1100)0- 

(1000)®! 
(0110) ff _ 


(0100) CT + <g> JB 

(llOO)*- 

(1000) CT _ 


[0]t 


S\B) 

(0010) CT _ ® B 

(2000)®+ 

(0200) ff + 

(0020) ff + 

(0000) CT + 


K Z {B) 

(0010) CT _ <g> s 
(2000),+ 
(0200) ff + 
(0000) CT + 



TABLE 1. Decomposition of the Hodge structures H*(X,8±) as 
representations of MT(9) SC = Sp(8,C) x Sp(10,C) 



irreducible representation of Sp(8, C) with highest weight a\UJ\ + • • • + 041^4 for 
the fundamental dominant weights u}±, . . . , Co>4, and let B be as in section |2] for 
(X, O). To indicate that a representation enters with multiplicity m > 1 we use 
a superscript ©m, and we specify the action of a = — idx with a a subscript 
a±. 

Lemma. The representation o/MT(9) sc = Sp(8, C) x Sp(10,C) on the Hodge 
structure M'(X,5±) is given in tabled with a = —idx acting trivially on B. 
Similarly, the representation of MT(C) = Sp(8, C) on M*(JC, p 5 a ) for a G 
{(5, 1), (4,2), (3,3)} is given in tablets 

Proof. This has been worked out using the computer algebra systems MAGMA 
and SAGE. For W(X, 8±) take the symmetric resp. alternating square of W(X, Sq) = 
(0000) [3} t © (1000) [2] t (0100) [l} t © ((0010) @B)) [0] 4 in the super sense and then 
subtract H*(X,t±). Here o acts by —1 on (1000) and on (0010) but trivially 
on (0000), (0100), (0001). To check that a acts trivially on B, note that by 
[BLl ex. 4.12(14)] the number of 2-torsion points on 9 is 2 9 ~ 1 (2 9 - 1) = 120; 
the Lefschetz fixed point formula for a then implies that o~*\b = ids- For the 
last three columns, for a > b > the Littlewood-Richardson rule in [WelJ says 
that p 5 a ^ is the difference of p 5 a * p 5b and p 5 a +i * p ^6-i up to complex shifts of Sjc- 
These complex shifts of 5jc can be computed as in the proof of lemma 13.51 □ 
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• 


U*(JC,P5 5A ) 


M'(JC,P5 4 , 2 ) 




[3]t 




(1000) 


(1000) 








(0010) 


[2]t 


(0000) 


(2000) 


(2000) 






(0100)® 2 


(1010) 






(0000)® 3 


(0100)® 2 








(0000) 


[1]* 


(1000)® 2 


(1100)®* 


(1100)® 2 






(1000)® 4 


(1000)® 3 






(0010) 


(0110) 








(0010) 


[0]* 


(2000) 


(2000) 


(2000)® 2 




(0100) 


(1010) 


(1010) 




(0000) 


(0200) 


(0200) 






(0100)® 3 


(0100) 






(0000)® 2 


(0020) 








(0000)® 2 



TABLE 2. Decomposition of the Hodge structures M'( JC, p 5 a ) as 
representations of MT(C) = Sp(8, C) 
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